Digital data processors operate in a serial, algorithmic mode and are capable of performing complex calculations very accurately and quickly. Such processors are incorporated in serial computers generally referred to as von Neumann type machines and they implement data manipulations in a step-by-step fashion. Many information processing problems can be solved by this approach, particularly those requiring repetitive calculations; however, von Neumann type computers perform inadequately when doing tasks involving pattern recognition, classification or associative learning. A further drawback of von Neumann type computers is presented by the fact that before a problem is amenable to solution, it must be fully understood and reduced to a series of algorithms and the algorithms must be translated into an appropriate language for processing by a particular computer. Construction of an appropriate algorithm for tasks involving interpretation of patterns, particularly dynamically changing patterns such as those encountered in speech recognition, high speed character recognition and interpretation of moving scenery present extremely difficult, if not impossible, tasks.
The brain of even a relatively simple organism represents a data processor operating in a parallel, distributed mode and it is capable of quickly and accurately interpreting a large body of dynamically changing data without the need for input of a complex algorithm. Such operation is even more impressive in view of the fact that signal propagation in the brain occurs at a speed many orders of magnitude lower than the speed of propagation of an electrical signal in a silicon chip. Biological neural systems are characterized by a very high degree of connectivity and signal processing is effected by both the degree and architecture of these connections as well as their ability to be altered and reconfigured by specific stimuli.
Investigations of biological systems have led to the development of neural computing networks also termed "parallel, distributed data processors." Such networks are characterized by the presence of a large number of individual computing elements typically termed "neurons," "unit cells," or "nodes." These individual cells are each interconnected to a plurality of other unit cells in a complex network. Connections between pairs of unit cells may be characterized as weak or strong and also as excitory or inhibitory. By the addition of the appropriate input and output circuitry to one or more neural processors, a neural network computer may be constructed. However, efforts made heretofore to construct neural network computers have led to complex software and have been hampered by inadequate hardware.
Neural network computing systems are trained to perform a particular task, rather than being programmed to execute an algorithm. Training is accomplished by configuring the pattern of connections between individual neurons. Training may be done in a passive mode by simply presetting the pattern, and in some instances the strength, of the connections between individual unit cells so as to elicit a desired response to a particular input. A more sophisticated approach involves a dynamic method wherein the actual output response of the network to a given input signal is correlated with a desired output response to generate a training signal which is then applied to the network to reconfigure the connections. A network of this type is able to "learn" an appropriate response to a given input stimulation. A dynamically trainable system can learn from its mistakes and is capable of a large degree of self-teaching.
While it is generally agreed that the massively parallel non-linear logic of neural network computers will readily adapt them to a wide variety of practical applications involving pattern recognition, speech synthesis and the solving of multi-parameter problems, the actual implementation of neural network information processing systems has been hampered by a lack of appropriate computing hardware. Presently, most investigations into neural network computing systems are carried out by emulating neural network systems on conventional von Neumann type computers. While such simulation allows for testing of particular neural network architectures, the conventional digital computer, operating in a serial manner, inherently presents a bottleneck to the parallel distributed processing approach of neural network systems. In some instances, dedicated computing systems comprised of a plurality of processors arranged in a parallel relationship have been utilized for neural network simulations. While these types of machines do confer some advantages in terms of speed, they do not provide true distributed processing and they still cannot simulate fully a large scale, highly interconnected, reconfigurable array of neurons. Furthermore, they are limited by the interconnect problem associated with increasing numbers of nodes, as will be explained more fully hereinbelow.
It is desirable to fabricate large scale, parallel, distributed data processors which comprise integrated arrays of interconnectable unit cells. The unit cells themselves are generally very simple devices for transferring data from one conductor to another, but the processor must be capable of establishing a complex pattern of interconnections therebetween. Two dimensional structures are not capable of providing a sufficiently large number of nodes to permit massively parallel, highly interconnected networks to be prepared; therefore three-dimensional structures are desired. Furthermore, computing power of the processor is greatly enhanced if the degree of connectivity between individual neurons may be controlled over a large dynamic range.
Heretofore, the art has not been adequate to enable the construction of large three-dimensional processing arrays of this type. If the switching of connections and the control of the degree of connectivity of a parallel distributed processor is implemented through the use of conventional semiconductor circuitry, the complexity of each unit cell increases significantly, thereby limiting the size and number of unit cells in a network. It would clearly be desirable to control the connection between individual unit cells through a simple, reliable circuit element which may be set to a range of values corresponding to different connectivities.
One attempt to provide a configurable neural network is disclosed by Thakoor et al in a Jet Propulsion Laboratory report numbered "JPLD-4166 (1987)" entitled "Content-Addressable High Density Memories Based on Neural Network Models." This approach relies upon an amorphous to crystalline transition first recognized by S. R. Ovshinsky, (see for example, "Reversible Electrical Switching Phenomena in Disordered Structures" Physical Review Letters V.21, N20, November 1968). The device of Thakoor et al. comprises a two-dimensional matrix of programmable amorphous silicon resistors interconnecting a series of simple unit cells. Each resistor is initially in a high resistivity state and may be set to a lower resistivity state by an appropriate pulse of current. By appropriately setting the resistors, the network is programmed; however, the resistors are not resettable hence, the system is not capable of being reconfigured or otherwise operating in a dynamic learning mode. Also, the resistors are not settable across a dynamic range of resistances and fine control of the degree of connectivity between interconnected cells is not possible.
Accordingly, it will be appreciated that there is a need for a simple neural network processor wherein the degree of connectivity between the unit cells may be simply and reliably set and reset in a cyclic, i.e. repeatable, mode. A system of this type is capable of a high degree of dynamic learning. It is further desirable that any such processor be adaptable to manufacture by standard device fabrication techniques. It is highly desirable that this network be structured as a large area, vertically interconnected three-dimensional device so as to increase processing density and decrease operational time.
While researchers have looked to the brain for initial inspiration in the development of neural network computing systems, they have continued to blindly rely upon conventional semiconductor structures and materials to implement these systems. Conventional semiconductor devices and materials operate in a volatile mode and are not well suited for neural circuitry. S. R. Ovshinsky has long recognized the fact that particular classes of materials can exhibit a range of physical properties which are analogous to those of biological neural systems. See, for example, "Analog Models for Information Storage and Transmission in Physiological Systems" by Stanford R. and Iris M. Ovshinsky in Mat. Res. Bull. Vol. 5, pp 681-690 (1970).
It has been found that certain materials, particularly chalcogenide,-based or containing, materials may be selectably, reversibly and cyclically set to a number of different values of a variety of physical properties such as electrical, optical, chemical, acoustic, pressure response and magnetic. In accord with the principles of the present invention these materials may be used as the basis for the interconnection of a number of unit cells into a three-dimensional neural network system. Materials of this type confer heretofore unattainable advantages in a neural network system insofar as they allow for ready programming and retraining of systems. Furthermore, these materials may be deposited in thin films over relatively large areas with high degrees of reliability and hence make possible the fabrication of large area, monolithic arrays of stacked unit cells and thereby provide a high density, massively parallel, distributed processing network.
The history of computing can be divided into phases based upon the problems presented by the interconnections of computing systems. The earliest electronic computers employed vacuum tubes and relays and the high failure rate, large power dissipation and bulk of these devices greatly restricted the number of nodes or connection points in these primitive systems. The development of the transistor made possible smaller, more reliable computing system thereby increasing the number of nodes and signal processing capabilities of such later computers. Integrated circuit technology made possible the inclusion of a very large number of transistors on a single chip and this greatly increased both the number of nodes and their degree of connectedness thereby making parallel processing systems possible.
The advent of parallel processing has raised the art to a point where a new barrier of connectivity has arisen which limits further development of such systems. Like the brain, realistic neuronal models should have nodes with unit dimensions on the order of square microns and should have a high degree of complex and reconfigurable interconnectivity. (See: S. R. Ovshinsky and I. M. Ovshinsky), "Analog Models for Information Storage and Transmission in Physiological Systems" Mat. Res. Bull. Vol. 5, pp 681-690 (1970, Pergamon Press)). Intelligence of neuronal systems is proportional to the number of nodes or neurons in the system as well as to the number of interconnections to which each node is a party. Furthermore, the information processing ability of the network is greatly increased if the nodes are interconnected in an analog manner whereby the strength of the interconnections may be varied over a range of values.
Through the present invention, previous problems of interconnectivity are overcome and it is now possible to provide a compact, highly interconnected neural network in which the connectivity between the various nodes may be adjusted and readjusted over a large dynamic range.
The computing systems provided by the present invention are true learning machines, unlike other parallel processors, insofar as they can adapt their connectivity to changing inputs in order to learn and can synthesize a creative output in response to novel stimuli. The systems of the present invention have utility in pattern recognition, adaptive control systems and in a wide variety of problem solving tasks.
These and other advantages of the present invention will be readily apparent from the drawings, discussion and description which follow.